<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
    "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
<head>
<meta http-equiv="Content-Type" content="application/xhtml+xml; charset=UTF-8" />
<meta name="generator" content="AsciiDoc 8.6.6" />
<title>GRR Developer documentation</title>
<style type="text/css">
/* Shared CSS for AsciiDoc xhtml11 and html5 backends */

/* Default font. */
body {
  font-family: Georgia,serif;
}

/* Title font. */
h1, h2, h3, h4, h5, h6,
div.title, caption.title,
thead, p.table.header,
#toctitle,
#author, #revnumber, #revdate, #revremark,
#footer {
  font-family: Arial,Helvetica,sans-serif;
}

body {
  margin: 1em 5% 1em 5%;
}

a {
  color: blue;
  text-decoration: underline;
}
a:visited {
  color: fuchsia;
}

em {
  font-style: italic;
  color: navy;
}

strong {
  font-weight: bold;
  color: #083194;
}

h1, h2, h3, h4, h5, h6 {
  color: #527bbd;
  margin-top: 1.2em;
  margin-bottom: 0.5em;
  line-height: 1.3;
}

h1, h2, h3 {
  border-bottom: 2px solid silver;
}
h2 {
  padding-top: 0.5em;
}
h3 {
  float: left;
}
h3 + * {
  clear: left;
}
h5 {
  font-size: 1.0em;
}

div.sectionbody {
  margin-left: 0;
}

hr {
  border: 1px solid silver;
}

p {
  margin-top: 0.5em;
  margin-bottom: 0.5em;
}

ul, ol, li > p {
  margin-top: 0;
}
ul > li     { color: #aaa; }
ul > li > * { color: black; }

pre {
  padding: 0;
  margin: 0;
}

#author {
  color: #527bbd;
  font-weight: bold;
  font-size: 1.1em;
}
#email {
}
#revnumber, #revdate, #revremark {
}

#footer {
  font-size: small;
  border-top: 2px solid silver;
  padding-top: 0.5em;
  margin-top: 4.0em;
}
#footer-text {
  float: left;
  padding-bottom: 0.5em;
}
#footer-badges {
  float: right;
  padding-bottom: 0.5em;
}

#preamble {
  margin-top: 1.5em;
  margin-bottom: 1.5em;
}
div.imageblock, div.exampleblock, div.verseblock,
div.quoteblock, div.literalblock, div.listingblock, div.sidebarblock,
div.admonitionblock {
  margin-top: 1.0em;
  margin-bottom: 1.5em;
}
div.admonitionblock {
  margin-top: 2.0em;
  margin-bottom: 2.0em;
  margin-right: 10%;
  color: #606060;
}

div.content { /* Block element content. */
  padding: 0;
}

/* Block element titles. */
div.title, caption.title {
  color: #527bbd;
  font-weight: bold;
  text-align: left;
  margin-top: 1.0em;
  margin-bottom: 0.5em;
}
div.title + * {
  margin-top: 0;
}

td div.title:first-child {
  margin-top: 0.0em;
}
div.content div.title:first-child {
  margin-top: 0.0em;
}
div.content + div.title {
  margin-top: 0.0em;
}

div.sidebarblock > div.content {
  background: #ffffee;
  border: 1px solid #dddddd;
  border-left: 4px solid #f0f0f0;
  padding: 0.5em;
}

div.listingblock > div.content {
  border: 1px solid #dddddd;
  border-left: 5px solid #f0f0f0;
  background: #f8f8f8;
  padding: 0.5em;
}

div.quoteblock, div.verseblock {
  padding-left: 1.0em;
  margin-left: 1.0em;
  margin-right: 10%;
  border-left: 5px solid #f0f0f0;
  color: #888;
}

div.quoteblock > div.attribution {
  padding-top: 0.5em;
  text-align: right;
}

div.verseblock > pre.content {
  font-family: inherit;
  font-size: inherit;
}
div.verseblock > div.attribution {
  padding-top: 0.75em;
  text-align: left;
}
/* DEPRECATED: Pre version 8.2.7 verse style literal block. */
div.verseblock + div.attribution {
  text-align: left;
}

div.admonitionblock .icon {
  vertical-align: top;
  font-size: 1.1em;
  font-weight: bold;
  text-decoration: underline;
  color: #527bbd;
  padding-right: 0.5em;
}
div.admonitionblock td.content {
  padding-left: 0.5em;
  border-left: 3px solid #dddddd;
}

div.exampleblock > div.content {
  border-left: 3px solid #dddddd;
  padding-left: 0.5em;
}

div.imageblock div.content { padding-left: 0; }
span.image img { border-style: none; }
a.image:visited { color: white; }

dl {
  margin-top: 0.8em;
  margin-bottom: 0.8em;
}
dt {
  margin-top: 0.5em;
  margin-bottom: 0;
  font-style: normal;
  color: navy;
}
dd > *:first-child {
  margin-top: 0.1em;
}

ul, ol {
    list-style-position: outside;
}
ol.arabic {
  list-style-type: decimal;
}
ol.loweralpha {
  list-style-type: lower-alpha;
}
ol.upperalpha {
  list-style-type: upper-alpha;
}
ol.lowerroman {
  list-style-type: lower-roman;
}
ol.upperroman {
  list-style-type: upper-roman;
}

div.compact ul, div.compact ol,
div.compact p, div.compact p,
div.compact div, div.compact div {
  margin-top: 0.1em;
  margin-bottom: 0.1em;
}

tfoot {
  font-weight: bold;
}
td > div.verse {
  white-space: pre;
}

div.hdlist {
  margin-top: 0.8em;
  margin-bottom: 0.8em;
}
div.hdlist tr {
  padding-bottom: 15px;
}
dt.hdlist1.strong, td.hdlist1.strong {
  font-weight: bold;
}
td.hdlist1 {
  vertical-align: top;
  font-style: normal;
  padding-right: 0.8em;
  color: navy;
}
td.hdlist2 {
  vertical-align: top;
}
div.hdlist.compact tr {
  margin: 0;
  padding-bottom: 0;
}

.comment {
  background: yellow;
}

.footnote, .footnoteref {
  font-size: 0.8em;
}

span.footnote, span.footnoteref {
  vertical-align: super;
}

#footnotes {
  margin: 20px 0 20px 0;
  padding: 7px 0 0 0;
}

#footnotes div.footnote {
  margin: 0 0 5px 0;
}

#footnotes hr {
  border: none;
  border-top: 1px solid silver;
  height: 1px;
  text-align: left;
  margin-left: 0;
  width: 20%;
  min-width: 100px;
}

div.colist td {
  padding-right: 0.5em;
  padding-bottom: 0.3em;
  vertical-align: top;
}
div.colist td img {
  margin-top: 0.3em;
}

@media print {
  #footer-badges { display: none; }
}

#toc {
  margin-bottom: 2.5em;
}

#toctitle {
  color: #527bbd;
  font-size: 1.1em;
  font-weight: bold;
  margin-top: 1.0em;
  margin-bottom: 0.1em;
}

div.toclevel1, div.toclevel2, div.toclevel3, div.toclevel4 {
  margin-top: 0;
  margin-bottom: 0;
}
div.toclevel2 {
  margin-left: 2em;
  font-size: 0.9em;
}
div.toclevel3 {
  margin-left: 4em;
  font-size: 0.9em;
}
div.toclevel4 {
  margin-left: 6em;
  font-size: 0.9em;
}

span.aqua { color: aqua; }
span.black { color: black; }
span.blue { color: blue; }
span.fuchsia { color: fuchsia; }
span.gray { color: gray; }
span.green { color: green; }
span.lime { color: lime; }
span.maroon { color: maroon; }
span.navy { color: navy; }
span.olive { color: olive; }
span.purple { color: purple; }
span.red { color: red; }
span.silver { color: silver; }
span.teal { color: teal; }
span.white { color: white; }
span.yellow { color: yellow; }

span.aqua-background { background: aqua; }
span.black-background { background: black; }
span.blue-background { background: blue; }
span.fuchsia-background { background: fuchsia; }
span.gray-background { background: gray; }
span.green-background { background: green; }
span.lime-background { background: lime; }
span.maroon-background { background: maroon; }
span.navy-background { background: navy; }
span.olive-background { background: olive; }
span.purple-background { background: purple; }
span.red-background { background: red; }
span.silver-background { background: silver; }
span.teal-background { background: teal; }
span.white-background { background: white; }
span.yellow-background { background: yellow; }

span.big { font-size: 2em; }
span.small { font-size: 0.6em; }

span.underline { text-decoration: underline; }
span.overline { text-decoration: overline; }
span.line-through { text-decoration: line-through; }


/*
 * xhtml11 specific
 *
 * */

tt {
  font-family: monospace;
  font-size: inherit;
  color: navy;
}

div.tableblock {
  margin-top: 1.0em;
  margin-bottom: 1.5em;
}
div.tableblock > table {
  border: 3px solid #527bbd;
}
thead, p.table.header {
  font-weight: bold;
  color: #527bbd;
}
p.table {
  margin-top: 0;
}
/* Because the table frame attribute is overriden by CSS in most browsers. */
div.tableblock > table[frame="void"] {
  border-style: none;
}
div.tableblock > table[frame="hsides"] {
  border-left-style: none;
  border-right-style: none;
}
div.tableblock > table[frame="vsides"] {
  border-top-style: none;
  border-bottom-style: none;
}


/*
 * html5 specific
 *
 * */

.monospaced {
  font-family: monospace;
  font-size: inherit;
  color: navy;
}

table.tableblock {
  margin-top: 1.0em;
  margin-bottom: 1.5em;
}
thead, p.tableblock.header {
  font-weight: bold;
  color: #527bbd;
}
p.tableblock {
  margin-top: 0;
}
table.tableblock {
  border-width: 3px;
  border-spacing: 0px;
  border-style: solid;
  border-color: #527bbd;
  border-collapse: collapse;
}
th.tableblock, td.tableblock {
  border-width: 1px;
  padding: 4px;
  border-style: solid;
  border-color: #527bbd;
}

table.tableblock.frame-topbot {
  border-left-style: hidden;
  border-right-style: hidden;
}
table.tableblock.frame-sides {
  border-top-style: hidden;
  border-bottom-style: hidden;
}
table.tableblock.frame-none {
  border-style: hidden;
}

th.tableblock.halign-left, td.tableblock.halign-left {
  text-align: left;
}
th.tableblock.halign-center, td.tableblock.halign-center {
  text-align: center;
}
th.tableblock.halign-right, td.tableblock.halign-right {
  text-align: right;
}

th.tableblock.valign-top, td.tableblock.valign-top {
  vertical-align: top;
}
th.tableblock.valign-middle, td.tableblock.valign-middle {
  vertical-align: middle;
}
th.tableblock.valign-bottom, td.tableblock.valign-bottom {
  vertical-align: bottom;
}


/*
 * manpage specific
 *
 * */

body.manpage h1 {
  padding-top: 0.5em;
  padding-bottom: 0.5em;
  border-top: 2px solid silver;
  border-bottom: 2px solid silver;
}
body.manpage h2 {
  border-style: none;
}
body.manpage div.sectionbody {
  margin-left: 3em;
}

@media print {
  body.manpage div#toc { display: none; }
}
@media screen {
  body {
    max-width: 50em; /* approximately 80 characters wide */
    margin-left: 16em;
  }

  #toc {
    position: fixed;
    top: 0;
    left: 0;
    bottom: 0;
    width: 13em;
    padding: 0.5em;
    padding-bottom: 1.5em;
    margin: 0;
    overflow: auto;
    border-right: 3px solid #f8f8f8;
    background-color: white;
  }

  #toc .toclevel1 {
    margin-top: 0.5em;
  }

  #toc .toclevel2 {
    margin-top: 0.25em;
    display: list-item;
    color: #aaaaaa;
  }

  #toctitle {
    margin-top: 0.5em;
  }
}
</style>
<script type="text/javascript">
/*<![CDATA[*/
var asciidoc = {  // Namespace.

/////////////////////////////////////////////////////////////////////
// Table Of Contents generator
/////////////////////////////////////////////////////////////////////

/* Author: Mihai Bazon, September 2002
 * http://students.infoiasi.ro/~mishoo
 *
 * Table Of Content generator
 * Version: 0.4
 *
 * Feel free to use this script under the terms of the GNU General Public
 * License, as long as you do not remove or alter this notice.
 */

 /* modified by Troy D. Hanson, September 2006. License: GPL */
 /* modified by Stuart Rackham, 2006, 2009. License: GPL */

// toclevels = 1..4.
toc: function (toclevels) {

  function getText(el) {
    var text = "";
    for (var i = el.firstChild; i != null; i = i.nextSibling) {
      if (i.nodeType == 3 /* Node.TEXT_NODE */) // IE doesn't speak constants.
        text += i.data;
      else if (i.firstChild != null)
        text += getText(i);
    }
    return text;
  }

  function TocEntry(el, text, toclevel) {
    this.element = el;
    this.text = text;
    this.toclevel = toclevel;
  }

  function tocEntries(el, toclevels) {
    var result = new Array;
    var re = new RegExp('[hH]([2-'+(toclevels+1)+'])');
    // Function that scans the DOM tree for header elements (the DOM2
    // nodeIterator API would be a better technique but not supported by all
    // browsers).
    var iterate = function (el) {
      for (var i = el.firstChild; i != null; i = i.nextSibling) {
        if (i.nodeType == 1 /* Node.ELEMENT_NODE */) {
          var mo = re.exec(i.tagName);
          if (mo && (i.getAttribute("class") || i.getAttribute("className")) != "float") {
            result[result.length] = new TocEntry(i, getText(i), mo[1]-1);
          }
          iterate(i);
        }
      }
    }
    iterate(el);
    return result;
  }

  var toc = document.getElementById("toc");
  if (!toc) {
    return;
  }

  // Delete existing TOC entries in case we're reloading the TOC.
  var tocEntriesToRemove = [];
  var i;
  for (i = 0; i < toc.childNodes.length; i++) {
    var entry = toc.childNodes[i];
    if (entry.nodeName == 'div'
     && entry.getAttribute("class")
     && entry.getAttribute("class").match(/^toclevel/))
      tocEntriesToRemove.push(entry);
  }
  for (i = 0; i < tocEntriesToRemove.length; i++) {
    toc.removeChild(tocEntriesToRemove[i]);
  }

  // Rebuild TOC entries.
  var entries = tocEntries(document.getElementById("content"), toclevels);
  for (var i = 0; i < entries.length; ++i) {
    var entry = entries[i];
    if (entry.element.id == "")
      entry.element.id = "_toc_" + i;
    var a = document.createElement("a");
    a.href = "#" + entry.element.id;
    a.appendChild(document.createTextNode(entry.text));
    var div = document.createElement("div");
    div.appendChild(a);
    div.className = "toclevel" + entry.toclevel;
    toc.appendChild(div);
  }
  if (entries.length == 0)
    toc.parentNode.removeChild(toc);
},


/////////////////////////////////////////////////////////////////////
// Footnotes generator
/////////////////////////////////////////////////////////////////////

/* Based on footnote generation code from:
 * http://www.brandspankingnew.net/archive/2005/07/format_footnote.html
 */

footnotes: function () {
  // Delete existing footnote entries in case we're reloading the footnodes.
  var i;
  var noteholder = document.getElementById("footnotes");
  if (!noteholder) {
    return;
  }
  var entriesToRemove = [];
  for (i = 0; i < noteholder.childNodes.length; i++) {
    var entry = noteholder.childNodes[i];
    if (entry.nodeName == 'div' && entry.getAttribute("class") == "footnote")
      entriesToRemove.push(entry);
  }
  for (i = 0; i < entriesToRemove.length; i++) {
    noteholder.removeChild(entriesToRemove[i]);
  }

  // Rebuild footnote entries.
  var cont = document.getElementById("content");
  var spans = cont.getElementsByTagName("span");
  var refs = {};
  var n = 0;
  for (i=0; i<spans.length; i++) {
    if (spans[i].className == "footnote") {
      n++;
      var note = spans[i].getAttribute("data-note");
      if (!note) {
        // Use [\s\S] in place of . so multi-line matches work.
        // Because JavaScript has no s (dotall) regex flag.
        note = spans[i].innerHTML.match(/\s*\[([\s\S]*)]\s*/)[1];
        spans[i].innerHTML =
          "[<a id='_footnoteref_" + n + "' href='#_footnote_" + n +
          "' title='View footnote' class='footnote'>" + n + "</a>]";
        spans[i].setAttribute("data-note", note);
      }
      noteholder.innerHTML +=
        "<div class='footnote' id='_footnote_" + n + "'>" +
        "<a href='#_footnoteref_" + n + "' title='Return to text'>" +
        n + "</a>. " + note + "</div>";
      var id =spans[i].getAttribute("id");
      if (id != null) refs["#"+id] = n;
    }
  }
  if (n == 0)
    noteholder.parentNode.removeChild(noteholder);
  else {
    // Process footnoterefs.
    for (i=0; i<spans.length; i++) {
      if (spans[i].className == "footnoteref") {
        var href = spans[i].getElementsByTagName("a")[0].getAttribute("href");
        href = href.match(/#.*/)[0];  // Because IE return full URL.
        n = refs[href];
        spans[i].innerHTML =
          "[<a href='#_footnote_" + n +
          "' title='View footnote' class='footnote'>" + n + "</a>]";
      }
    }
  }
},

install: function(toclevels) {
  var timerId;

  function reinstall() {
    asciidoc.footnotes();
    if (toclevels) {
      asciidoc.toc(toclevels);
    }
  }

  function reinstallAndRemoveTimer() {
    clearInterval(timerId);
    reinstall();
  }

  timerId = setInterval(reinstall, 500);
  if (document.addEventListener)
    document.addEventListener("DOMContentLoaded", reinstallAndRemoveTimer, false);
  else
    window.onload = reinstallAndRemoveTimer;
}

}
asciidoc.install(2);
/*]]>*/
</script>
</head>
<body class="article">
<div id="header">
<h1>GRR Developer documentation</h1>
<div id="toc">
  <div id="toctitle">Table of Contents</div>
  <noscript><p><b>JavaScript must be enabled in your browser to display the table of contents.</b></p></noscript>
</div>
</div>
<div id="content">
<div id="preamble">
<div class="sectionbody">
<div class="paragraph"><p>GRR is forensic framework focused on scalability enabling powerful analysis. To
achieve this it has a lot of moving parts and a rich API. This document examines
each subsystem and explains how they interact. We then proceed to examine the
implementation from a developer&#8217;s point of view, to assist external contributors
to come up to speed with the code-base.</p></div>
</div>
</div>
<div class="sect1">
<h2 id="_grr_component_overview">GRR Component Overview</h2>
<div class="sectionbody">
<div class="paragraph"><p>The following describes the components that make up the GRR system. Since GRR is
designed to be scalable, each component can exist on its own machine in the data
center. Alternatively, for smaller installations, each component can run as a
separate server process, or even all components can run inside the same process
in different threads.</p></div>
<div class="imageblock" style="text-align:center;">
<div class="content">
<img src="images/overview.png" alt="images/overview.png" width="600" />
</div>
<div class="title">Figure 1: GRR Component overview</div>
</div>
<div class="paragraph"><p>The main components running components are:</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
Client
</dt>
<dd>
<p>
The GRR client agent is deployed on corporate assets using the usual mechanism
for software distribution and updates (e.g. SMS, apt). The client communicates
with the front-end server using a HTTP POST request. The request sends and
receives <em>GRR messages</em> from the server (more detail on this below). All
communication with the front end servers is encrypted.
</p>
</dd>
<dt class="hdlist1">
Front End Servers
</dt>
<dd>
<p>
The front end servers' main task is to decrypt POST requests from the client,
un-bundle the contained messages and queues these on the data store. The front
end also fetches any messages queued for the client and sends them to the
client.
</p>
</dd>
<dt class="hdlist1">
Datastore
</dt>
<dd>
<p>
The data store acts both as a central storage component for data, and as a
communication mechanism for all the components. The data is modeled after an RDF
datastore using the AFF4 data model (See <a href="#datamodel">AFF4 Data Model</a>).
</p>
</dd>
<dt class="hdlist1">
Console
</dt>
<dd>
<p>
The console is the central application which enables the incident responder or
forensic analyst to interact with the system. The consoles allows for analysis
tasks to be queued for the clients, and results of previous stored analysis to
be examined. GRR has both a GUI and a text based console application.
</p>
</dd>
<dt class="hdlist1">
Worker
</dt>
<dd>
<p>
In order to remain scalable, the front end does not do any processing of data,
preferring to offload processing to special worker components. The number of
workers can be tuned in response to increased workload. Workers typically check
queues in the data stores for responses from the client, process those and
re-queue new requests for the clients (See <a href="#Flows">[Flows]</a> and <a href="#Queues">[Queues]</a>).
</p>
</dd>
</dl></div>
</div>
</div>
<div class="sect1">
<h2 id="_grr_messages">GRR Messages</h2>
<div class="sectionbody">
<div class="paragraph"><p>On the wire, the client and server interchange messages. We term the messages
send from server to the client <em>requests</em>, while messages send from the client
to the server are <em>responses</em>.  Requests sent to the client, ask the client to
perform some action, which we term the <em>client action</em>, for example
<em>ListDirectory</em>. A single request may elicit multiple responses.</p></div>
<div class="paragraph"><p>For example, the request message <em>ListDirectory</em> will elicit a response for each
file in the directory (potentially thousands). Requests and responses are
tracked using an incremental <em>request_id</em> and <em>response_id</em>.</p></div>
<div class="paragraph"><p>In order to indicate when all responses have been sent for a particular request,
the client sends a special STATUS message as the last response. In the event of
errors, the message contains details of the error (including backtrace). If the
action completed successfully, an OK status is returned.</p></div>
<div class="paragraph"><p>Messages are encoded as GrrMessage protobufs:</p></div>
<div class="tableblock">
<table rules="all"
width="100%"
frame="border"
cellspacing="0" cellpadding="4">
<caption class="title">Table 1. Important GRR Message protobuf fields.</caption>
<col width="50%" />
<col width="50%" />
<tbody>
<tr>
<td align="left" valign="top"><p class="table">session_id</p></td>
<td align="left" valign="top"><p class="table">A unique integer for all packets generated by this flow.</p></td>
</tr>
<tr>
<td align="left" valign="top"><p class="table">name</p></td>
<td align="left" valign="top"><p class="table">The name of the Action to be called on the client (See below).</p></td>
</tr>
<tr>
<td align="left" valign="top"><p class="table">args</p></td>
<td align="left" valign="top"><p class="table">A serialized protobuf which will be interpreted by the Action.</p></td>
</tr>
<tr>
<td align="left" valign="top"><p class="table">request_id</p></td>
<td align="left" valign="top"><p class="table">An incrementing number of this request (see below)</p></td>
</tr>
<tr>
<td align="left" valign="top"><p class="table">response_id</p></td>
<td align="left" valign="top"><p class="table">An incrementing number of the response (see below)</p></td>
</tr>
</tbody>
</table>
</div>
<div class="imageblock" style="text-align:center;">
<div class="content">
<img src="images/messages.png" alt="images/messages.png" width="600" />
</div>
<div class="title">Figure 2: Typical Message Request/Response Sequence.</div>
</div>
<div class="paragraph"><p>Figure 2 illustrates a typical sequence of messages. Request 1 was sent from the
server to the client, and elicited 3 responses, in addition to a status message.</p></div>
<div class="paragraph"><p>When the server sends the client messages, the messages are tagged in the data
store with a lease time. If the client does not reply for these requests within
the lease time, the requests become available for lease again. This is designed
for the case of the client rebooting or losing connectivity part way through
running the action. In this case, the request is re-transmitted and the action
is run again.</p></div>
<div class="sect2">
<h3 id="_client_communication_and_encryption">Client Communication and Encryption</h3>
<div class="paragraph"><p>TBD</p></div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="Flows">Flows.</h2>
<div class="sectionbody">
<div class="paragraph"><p>Typically when we want to run an analysis on the client, we need to go through a
process where a number of requests are issued, their responses are examined, and
new requests are issued. This round trip to the client may happen many times.</p></div>
<div class="paragraph"><p>If it inefficient to dedicate server resources to wait for the client to
complete any particular request. The client may lose connectivity or disappear
at any time, and leave server resources (such as memory) in use for prolonged
period.</p></div>
<div class="paragraph"><p>GRR solves the problem by using a suspendable execution model. We term this
construct a <em>Flow</em>. The flow is a state machine which receives responses to
requests into well defined <em>state methods</em>. The flow object can be pickled
(serialized) between state executions to disk. This allows the server to suspect
the current execution flow with the client indefinitely with no server resources
committed.</p></div>
<div class="imageblock" style="text-align:center;">
<div class="content">
<img src="images/flow.png" alt="images/flow.png" width="600" />
</div>
<div class="title">Figure 4: An example flow to fetch a file from the client.</div>
</div>
<div class="paragraph"><p>Figure 4 illustrates a typical flow:</p></div>
<div class="olist arabic"><ol class="arabic">
<li>
<p>
We begin in the <em>Start</em> state, where we issue a request to the client to
calculate the hash of a file. The responses to the request (when they arrive
back from the client) will be fed into the <em>CheckHash</em> state. Once the requests
are send, the flow can be serialized indefinitely.
</p>
</li>
<li>
<p>
When the hash response arrives from the client, and we do not have the hash
in the database (i.e. we do not have a copy of this file), the flow will issue a
<em>ReadBuffer</em> request for the first buffer of the file.
</p>
</li>
<li>
<p>
When the response is returned, the buffer is written to disk, and the next
buffer is requested.
</p>
</li>
<li>
<p>
Once the entire length of the file is retrieved, we can verify the buffer
hash and complete the flow.
</p>
</li>
</ol></div>
<div class="admonitionblock">
<table><tr>
<td class="icon">
<img src="./images/icons/note.png" alt="Note" />
</td>
<td class="content">
<div class="paragraph"><p>The above flow is rather inefficient since it requires a round trip for each
buffer fetched. In practice, the <em>GetFile</em> flow will queue up a full window of
<em>ReadBuffer</em> requests in order to make better use of client bandwidth.</p></div>
<div class="paragraph"><p>When designing real flows, it is always necessary to minimize round trips.</p></div>
</td>
</tr></table>
</div>
<div class="paragraph"><p>Since multiple flows can run simultaneously on the same client, it is necessary
to be able to distinguish between flows. Flows have a <em>session_id</em> which is a
randomly generated ID to designate all requests and responses belonging to this
flow. The client simply copies the session id from requests to any responses it
generates, to ensure they will arrive at the respective flow. For example, a
session id may be &#8216;W:1213243AE&#8217;.</p></div>
<div class="sect2">
<h3 id="_well_known_flows_and_the_event_system">Well Known Flows and the Event System</h3>
<div class="paragraph"><p>The GRR client receives requests with a flow session id, which it copies to the
response. This session id is used to route the response to the correct
flow. This means that typically, the client is unable to just send a message
into any running flow, since it would require guessing the session id for the
flow. This is a good thing since it protects flows from malicious clients.</p></div>
<div class="paragraph"><p>However, there is a need sometimes for the client to send unsolicited messages
to the server (For example during enrollment). In order for the responses to be
routed they must have a correct session id.</p></div>
<div class="paragraph"><p>The solution is to have a second class of flows on the server called <em>Well Known
Flows</em>. These flows have a well known session id so messages can be routed to
them without the flow needing to send requests for them first. Since any client
or server can send messages to a well known flow, it is impossible for these to
maintain state (since its the same instance of the flow used to process many
clients' messages. Therefore well known flows are stateless.</p></div>
<div class="paragraph"><p>A well known flow typically is used simply to launch another flow or to process
an event. For example:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight 3.1.5
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt><span style="font-weight: bold"><span style="color: #0000FF">class</span></span> <span style="font-weight: bold"><span style="color: #000000">Enroler</span></span><span style="color: #990000">(</span>flow<span style="color: #990000">.</span>WellKnownFlow<span style="color: #990000">):</span>                                      <span style="font-style: italic"><span style="color: #9A1900"># <img src="./images/icons/callouts/1.png" alt="1" /></span></span>
<span style="font-style: italic"><span style="color: #9A1900">  """Manage enrollment requests."""</span></span>
  well_known_session_id <span style="color: #990000">=</span> <span style="color: #FF0000">"CA:Enrol"</span>                                    <span style="font-style: italic"><span style="color: #9A1900"># <img src="./images/icons/callouts/2.png" alt="2" /></span></span>

  <span style="font-weight: bold"><span style="color: #0000FF">def</span></span> <span style="font-weight: bold"><span style="color: #000000">ProcessMessage</span></span><span style="color: #990000">(</span>self<span style="color: #990000">,</span> message<span style="color: #990000">):</span>
<span style="font-style: italic"><span style="color: #9A1900">    """Begins an enrollment flow for this client."""</span></span>
    cert <span style="color: #990000">=</span> jobs_pb2<span style="color: #990000">.</span><span style="font-weight: bold"><span style="color: #000000">Certificate</span></span><span style="color: #990000">()</span>                                       <span style="font-style: italic"><span style="color: #9A1900"># <img src="./images/icons/callouts/3.png" alt="3" /></span></span>
    cert<span style="color: #990000">.</span><span style="font-weight: bold"><span style="color: #000000">ParseFromString</span></span><span style="color: #990000">(</span>message<span style="color: #990000">.</span>args<span style="color: #990000">)</span>

    client_id <span style="color: #990000">=</span> message<span style="color: #990000">.</span>source

    <span style="font-style: italic"><span style="color: #9A1900"># Create a new client object for this client.</span></span>
    client <span style="color: #990000">=</span> aff4<span style="color: #990000">.</span>FACTORY<span style="color: #990000">.</span><span style="font-weight: bold"><span style="color: #000000">Create</span></span><span style="color: #990000">(</span>client_id<span style="color: #990000">,</span> <span style="color: #FF0000">"VFSGRRClient"</span><span style="color: #990000">,</span> mode<span style="color: #990000">=</span><span style="color: #FF0000">"rw"</span><span style="color: #990000">,</span>  <span style="font-style: italic"><span style="color: #9A1900"># <img src="./images/icons/callouts/4.png" alt="4" /></span></span>
                                 token<span style="color: #990000">=</span>self<span style="color: #990000">.</span>token<span style="color: #990000">)</span>

    <span style="font-style: italic"><span style="color: #9A1900"># Only enroll this client if it has no certificate yet.</span></span>
    <span style="font-weight: bold"><span style="color: #0000FF">if</span></span> <span style="font-weight: bold"><span style="color: #0000FF">not</span></span> client<span style="color: #990000">.</span><span style="font-weight: bold"><span style="color: #000000">Get</span></span><span style="color: #990000">(</span>client<span style="color: #990000">.</span>Schema<span style="color: #990000">.</span>CERT<span style="color: #990000">):</span>
      <span style="font-style: italic"><span style="color: #9A1900"># Start the enrollment flow for this client.</span></span>
      flow<span style="color: #990000">.</span>FACTORY<span style="color: #990000">.</span><span style="font-weight: bold"><span style="color: #000000">StartFlow</span></span><span style="color: #990000">(</span>client_id<span style="color: #990000">=</span>client_id<span style="color: #990000">,</span> flow_name<span style="color: #990000">=</span><span style="color: #FF0000">"CAEnroler"</span><span style="color: #990000">,</span><span style="font-style: italic"><span style="color: #9A1900"># <img src="./images/icons/callouts/5.png" alt="5" /></span></span>
                             csr<span style="color: #990000">=</span>cert<span style="color: #990000">,</span> queue_name<span style="color: #990000">=</span><span style="color: #FF0000">"CA"</span><span style="color: #990000">,</span>
                             _client<span style="color: #990000">=</span>client<span style="color: #990000">,</span> token<span style="color: #990000">=</span>self<span style="color: #990000">.</span>token<span style="color: #990000">)</span></tt></pre></div></div>
<div class="colist arabic"><table>
<tr><td><img src="./images/icons/callouts/1.png" alt="1" /></td><td>
A new well known flow is created by extending the WellKnownFlow baseclass.
</td></tr>
<tr><td><img src="./images/icons/callouts/2.png" alt="2" /></td><td>
The Well known session id of this flow is "CA:Enrol". Messages to this queue
will be routed to this flow.
</td></tr>
<tr><td><img src="./images/icons/callouts/3.png" alt="3" /></td><td>
We expect to receive a Certificate protobuf for this flow - this is
essentially a CSR for the requesting client.
</td></tr>
<tr><td><img src="./images/icons/callouts/4.png" alt="4" /></td><td>
We open the AFF4 object for this client.
</td></tr>
<tr><td><img src="./images/icons/callouts/5.png" alt="5" /></td><td>
If this client has no certificate already, we launch the real CAEnroler flow
to talk with this client. Note that the CAEnroler flow is a regular flow with a
random session_id - that ensure no other clients can interfere with this
enrollment request.
</td></tr>
</table></div>
<div class="paragraph"><p>On top of the the Well known flow system GRR implements an event dispatching
system. This allows any component to declare an event it is interested in, and
have any other component broadcast the event.</p></div>
<div class="paragraph"><p>For example, when a new client enrolls we wish to run the Interrogate flow
automatically for it (so we can query it for hostname, usernames etc):</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight 3.1.5
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt><span style="font-weight: bold"><span style="color: #0000FF">class</span></span> <span style="font-weight: bold"><span style="color: #000000">EnrolmentInterrogateEvent</span></span><span style="color: #990000">(</span>flow<span style="color: #990000">.</span>EventListener<span style="color: #990000">):</span>                  <span style="font-style: italic"><span style="color: #9A1900"># <img src="./images/icons/callouts/1.png" alt="1" /></span></span>
<span style="font-style: italic"><span style="color: #9A1900">  """An event handler which will schedule interrogation on client enrollment."""</span></span>
  EVENTS <span style="color: #990000">=</span> <span style="color: #990000">[</span><span style="color: #FF0000">"ClientEnrollment"</span><span style="color: #990000">]</span>                                       <span style="font-style: italic"><span style="color: #9A1900"># <img src="./images/icons/callouts/2.png" alt="2" /></span></span>
  well_known_session_id <span style="color: #990000">=</span> <span style="color: #FF0000">"W:Interrogate"</span>

  @flow<span style="color: #990000">.</span><span style="font-weight: bold"><span style="color: #000000">EventHandler</span></span><span style="color: #990000">(</span>source_restriction<span style="color: #990000">=</span><span style="color: #FF0000">"CA"</span><span style="color: #990000">)</span>
  <span style="font-weight: bold"><span style="color: #0000FF">def</span></span> <span style="font-weight: bold"><span style="color: #000000">ProcessMessage</span></span><span style="color: #990000">(</span>self<span style="color: #990000">,</span> message<span style="color: #990000">=</span>None<span style="color: #990000">,</span> event<span style="color: #990000">=</span>None<span style="color: #990000">):</span>
    flow<span style="color: #990000">.</span>FACTORY<span style="color: #990000">.</span><span style="font-weight: bold"><span style="color: #000000">StartFlow</span></span><span style="color: #990000">(</span>event<span style="color: #990000">.</span>cn<span style="color: #990000">,</span> <span style="color: #FF0000">"Interrogate"</span><span style="color: #990000">,</span> token<span style="color: #990000">=</span>self<span style="color: #990000">.</span>token<span style="color: #990000">)</span> <span style="font-style: italic"><span style="color: #9A1900"># <img src="./images/icons/callouts/3.png" alt="3" /></span></span></tt></pre></div></div>
<div class="colist arabic"><table>
<tr><td><img src="./images/icons/callouts/1.png" alt="1" /></td><td>
An event listener is a class which extends flow.EventListener.
</td></tr>
<tr><td><img src="./images/icons/callouts/2.png" alt="2" /></td><td>
It will listen to any of these events. Note we also need to have a unique
well_known_session_id since an Event Listener is just a specialized Well Known
Flow.
</td></tr>
<tr><td><img src="./images/icons/callouts/3.png" alt="3" /></td><td>
When an event called "ClientEnrollment" is broadcast, this EventListener
will receive the message and automatically run the Interrogate flow against this
client.
</td></tr>
</table></div>
<div class="paragraph"><p>The event can now be broadcast from any other flow:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight 3.1.5
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt><span style="font-weight: bold"><span style="color: #0000FF">class</span></span> <span style="font-weight: bold"><span style="color: #000000">CAEnroler</span></span><span style="color: #990000">(</span>flow<span style="color: #990000">.</span>GRRFlow<span style="color: #990000">):</span>
<span style="font-style: italic"><span style="color: #9A1900">  """Enrol new clients."""</span></span>
  <span style="color: #990000">.....</span>

  @flow<span style="color: #990000">.</span><span style="font-weight: bold"><span style="color: #000000">StateHandler</span></span><span style="color: #990000">(</span>next_state<span style="color: #990000">=</span><span style="color: #FF0000">"End"</span><span style="color: #990000">)</span>
  <span style="font-weight: bold"><span style="color: #0000FF">def</span></span> <span style="font-weight: bold"><span style="color: #000000">Start</span></span><span style="color: #990000">(</span>self<span style="color: #990000">):</span>
    <span style="color: #990000">.....</span>
    <span style="font-style: italic"><span style="color: #9A1900"># Publish the client enrollment message.</span></span>
    self<span style="color: #990000">.</span><span style="font-weight: bold"><span style="color: #000000">Publish</span></span><span style="color: #990000">(</span><span style="color: #FF0000">"ClientEnrollment"</span><span style="color: #990000">,</span> certificate_attribute<span style="color: #990000">.</span><span style="font-weight: bold"><span style="color: #000000">AsProto</span></span><span style="color: #990000">())</span></tt></pre></div></div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="Hunts">Hunts</h2>
<div class="sectionbody">
<div class="paragraph"><p>In order to be able to search for malicious code and abnormal behavior
amongst the entire fleet of clients, GRR has implemented the concept
of hunts. Hunts are essentially flows that run on multiple clients at
the same time.</p></div>
<div class="paragraph"><p>At the moment, hunts are regarded as rather high impact since they can
affect many clients at once so scheduling new hunts is restricted to
console users. In order to schedule a hunt, this is the process:</p></div>
<div class="paragraph"><p>Import the hunts file and create a new hunt (This example uses the
SampleHunt, replace with a hunt of your choice):</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight 3.1.5
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>hunt <span style="color: #990000">=</span> hunts<span style="color: #990000">.</span><span style="font-weight: bold"><span style="color: #000000">SampleHunt</span></span><span style="color: #990000">(</span>client_limit<span style="color: #990000">=</span><span style="color: #993399">100</span><span style="color: #990000">,</span> expiry_time<span style="color: #990000">=</span><span style="color: #993399">36000</span><span style="color: #990000">)</span></tt></pre></div></div>
<div class="paragraph"><p>This creates the SampleHunt, possible parameters:</p></div>
<div class="ulist"><ul>
<li>
<p>
client_limit to limit the number of clients this hunt will be
  scheduled on for experiments.
</p>
</li>
<li>
<p>
expiry_time gives the time this hunt will be active (in seconds).
</p>
</li>
</ul></div>
<div class="paragraph"><p>Next you have to create rules to indicate which clients the hunt will
be run on. Rules either compare fields stored in the client
description to integer values or match regular expressions against
them:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight 3.1.5
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>int_rule <span style="color: #990000">=</span> jobs_pb2<span style="color: #990000">.</span><span style="font-weight: bold"><span style="color: #000000">ForemanAttributeInteger</span></span><span style="color: #990000">(</span>
                   attribute_name<span style="color: #990000">=</span>client<span style="color: #990000">.</span>Schema<span style="color: #990000">.</span>OS_RELEASE<span style="color: #990000">.</span>name<span style="color: #990000">,</span>
                   operator<span style="color: #990000">=</span>jobs_pb2<span style="color: #990000">.</span>ForemanAttributeInteger<span style="color: #990000">.</span>EQUAL<span style="color: #990000">,</span>
                   value<span style="color: #990000">=</span><span style="color: #993399">7</span><span style="color: #990000">)</span>
regex_rule <span style="color: #990000">=</span> flow<span style="color: #990000">.</span>GRRHunt<span style="color: #990000">.</span>MATCH_WINDOWS</tt></pre></div></div>
<div class="paragraph"><p>The two rules above will match clients that run windows and have an
OS_RELEASE of 7.</p></div>
<div class="paragraph"><p>Next, you add the rules to the hunt:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight 3.1.5
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>hunt<span style="color: #990000">.</span><span style="font-weight: bold"><span style="color: #000000">AddRule</span></span><span style="color: #990000">([</span>int_rule<span style="color: #990000">,</span> regex_rule<span style="color: #990000">])</span></tt></pre></div></div>
<div class="paragraph"><p>You can call AddRule more than once. The hunt will be triggered if all
the rules of one AddRule call match at the same time. So you can do:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight 3.1.5
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>hunt<span style="color: #990000">.</span><span style="font-weight: bold"><span style="color: #000000">AddRule</span></span><span style="color: #990000">([</span> flow<span style="color: #990000">.</span>GRRHunt<span style="color: #990000">.</span>MATCH_WINDOWS<span style="color: #990000">,</span>
  jobs_pb2<span style="color: #990000">.</span><span style="font-weight: bold"><span style="color: #000000">ForemanAttributeInteger</span></span><span style="color: #990000">(</span>attribute_name<span style="color: #990000">=</span>client<span style="color: #990000">.</span>Schema<span style="color: #990000">.</span>OS_RELEASE<span style="color: #990000">.</span>name<span style="color: #990000">,</span>
  operator<span style="color: #990000">=</span>jobs_pb2<span style="color: #990000">.</span>ForemanAttributeInteger<span style="color: #990000">.</span>EQUAL<span style="color: #990000">,</span> value<span style="color: #990000">=</span><span style="color: #993399">7</span><span style="color: #990000">)])</span>

hunt<span style="color: #990000">.</span><span style="font-weight: bold"><span style="color: #000000">AddRule</span></span><span style="color: #990000">([</span> flow<span style="color: #990000">.</span>GRRHunt<span style="color: #990000">.</span>MATCH_WINDOWS<span style="color: #990000">,</span>
  jobs_pb2<span style="color: #990000">.</span><span style="font-weight: bold"><span style="color: #000000">ForemanAttributeInteger</span></span><span style="color: #990000">(</span>attribute_name<span style="color: #990000">=</span>client<span style="color: #990000">.</span>Schema<span style="color: #990000">.</span>OS_RELEASE<span style="color: #990000">.</span>name<span style="color: #990000">,</span>
  operator<span style="color: #990000">=</span>jobs_pb2<span style="color: #990000">.</span>ForemanAttributeInteger<span style="color: #990000">.</span>EQUAL<span style="color: #990000">,</span> value<span style="color: #990000">=</span><span style="color: #993399">8</span><span style="color: #990000">)])</span></tt></pre></div></div>
<div class="paragraph"><p>to start a hunt on windows clients that have either OS_VERSION 7 or 8.</p></div>
<div class="paragraph"><p>If you are unsure about your rules, you can call</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight 3.1.5
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>hunt<span style="color: #990000">.</span><span style="font-weight: bold"><span style="color: #000000">TestRules</span></span><span style="color: #990000">()</span></tt></pre></div></div>
<div class="paragraph"><p>This will show you how many clients in the db match your rules and
give you some matching sample clients. Note that this will open all
the clients and will therefore take a while if you have a lot of
clients in the database.</p></div>
<div class="paragraph"><p>Once you are happy with your rules, you upload the hunt to the
foreman:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight 3.1.5
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>hunt<span style="color: #990000">.</span><span style="font-weight: bold"><span style="color: #000000">Run</span></span><span style="color: #990000">()</span></tt></pre></div></div>
<div class="paragraph"><p>At this point, matching clients will pick up and run the hunt and you
can see the progress in the UI.</p></div>
<div class="paragraph"><p>If you want to stop a running hunt and remove the foreman rules again, use</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight 3.1.5
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>hunt<span style="color: #990000">.</span><span style="font-weight: bold"><span style="color: #000000">Stop</span></span><span style="color: #990000">()</span></tt></pre></div></div>
</div>
</div>
<div class="sect1">
<h2 id="Queues">Queues</h2>
<div class="sectionbody">
<div class="paragraph"><p>GRR Supports multiple processing queues on the server. This allows specialized
workers to be used for specific tasks, independently routing all messages to
these workers.</p></div>
<div class="imageblock" style="text-align:center;">
<div class="content">
<img src="images/queues.png" alt="images/queues.png" width="400" />
</div>
<div class="title">Figure 3: Processing queues in the GRR architecture.</div>
</div>
<div class="paragraph"><p>For example Figure 3 illustrates three distinct types of workers. The general
purpose workers retrieve messages from the general queue named "W". A
specialized worker responsible for CA enrollments communicates to the client on
the queue "CA" while an interactive worker has its own queue for the client.</p></div>
<div class="paragraph"><p>It is also possible to run special purpose workers for example for debugging or
special flow processing (i.e. workers with custom flows which are not found in
the standard workers). These workers can be started with the command line
parameter <em>--worker_queue_name</em> to specify a custom name. The special worker
will then attach to the regular messaging system and be able to issue flows to
the client without interference from other workers.</p></div>
<div class="paragraph"><p>Additionally each client has a queue for messages intended to it (i.e. <em>client
requests</em>). This queue can be examined using the <em>Debug Client Requests</em> option
in the GUI:</p></div>
<div class="imageblock">
<div class="content">
<img src="images/client_requests.jpg" alt="images/client_requests.jpg" />
</div>
<div class="title">Figure 1. Inspecting outstanding client requests.</div>
</div>
<div class="paragraph"><p>The Figure shows three client requests outstanding for this client. The client
is currently offline and so has requests queued for it when it returns. We can
see two regular requests directed to the queue "W" - a ListProcesses request and
a NetStat request. There is also a special request directed at the queue DEBUG
for ExecuteBinaryRequest.</p></div>
</div>
</div>
<div class="sect1">
<h2 id="datamodel">AFF4 Data Model</h2>
<div class="sectionbody">
<div class="paragraph"><p>AFF4 was first published in 2008 as an extensible, modern forensic storage
format. The AFF4 data model allows the representation of arbitrary objects and
the association of these with semantic meaning. The AFF4 data model is at the
heart of GRR and is essential for understanding how GRR store, analyzes and
represents forensic artifacts.</p></div>
<div class="paragraph"><p>AFF4 is an object oriented model. This means that all entities are just
different types of <em>AFF4 objects</em>. An AFF4 object is simply an entity,
addressable by a globally unique name, which has attributes attached to it, as
well as behaviors.</p></div>
<div class="paragraph"><p>Each AFF4 object has a unique urn by which it can be addressed. AFF4 objects
also have optional attributes which are defined in the object&#8217;s Schema. For
example consider the following definition of an AFF4 object representing a GRR
Client:</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight 3.1.5
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt><span style="font-weight: bold"><span style="color: #0000FF">class</span></span> <span style="font-weight: bold"><span style="color: #000000">VFSGRRClient</span></span><span style="color: #990000">(</span>aff4<span style="color: #990000">.</span>AFF4Object<span style="color: #990000">):</span>                                   <span style="font-style: italic"><span style="color: #9A1900"># <img src="./images/icons/callouts/1.png" alt="1" /></span></span>
<span style="font-style: italic"><span style="color: #9A1900">  """A Remote client."""</span></span>

  <span style="font-weight: bold"><span style="color: #0000FF">class</span></span> <span style="font-weight: bold"><span style="color: #000000">SchemaCls</span></span><span style="color: #990000">(</span>aff4<span style="color: #990000">.</span>AFF4Object<span style="color: #990000">.</span>SchemaCls<span style="color: #990000">):</span>                          <span style="font-style: italic"><span style="color: #9A1900"># <img src="./images/icons/callouts/2.png" alt="2" /></span></span>
<span style="font-style: italic"><span style="color: #9A1900">    """The schema for the client."""</span></span>
    CERT <span style="color: #990000">=</span> aff4<span style="color: #990000">.</span><span style="font-weight: bold"><span style="color: #000000">Attribute</span></span><span style="color: #990000">(</span><span style="color: #FF0000">"metadata:cert"</span><span style="color: #990000">,</span> RDFX509Cert<span style="color: #990000">,</span>                <span style="font-style: italic"><span style="color: #9A1900"># <img src="./images/icons/callouts/3.png" alt="3" /></span></span>
                          <span style="color: #FF0000">"The PEM encoded cert of the client."</span><span style="color: #990000">)</span>

    <span style="font-style: italic"><span style="color: #9A1900"># Information about the host.</span></span>
    HOSTNAME <span style="color: #990000">=</span> aff4<span style="color: #990000">.</span><span style="font-weight: bold"><span style="color: #000000">Attribute</span></span><span style="color: #990000">(</span><span style="color: #FF0000">"metadata:hostname"</span><span style="color: #990000">,</span> aff4<span style="color: #990000">.</span>RDFString<span style="color: #990000">,</span>     <span style="font-style: italic"><span style="color: #9A1900"># <img src="./images/icons/callouts/4.png" alt="4" /></span></span>
                              <span style="color: #FF0000">"Hostname of the host."</span><span style="color: #990000">,</span> <span style="color: #FF0000">"Host"</span><span style="color: #990000">,</span>
                              index<span style="color: #990000">=</span>client_index<span style="color: #990000">)</span></tt></pre></div></div>
<div class="colist arabic"><table>
<tr><td><img src="./images/icons/callouts/1.png" alt="1" /></td><td>
An AFF4 object is simply a class which extends the AFF4Object base class.
</td></tr>
<tr><td><img src="./images/icons/callouts/2.png" alt="2" /></td><td>
Each AFF4 object contains a Schema - in this case the Schema extends the
base AFF4Object schema - this means this object can contains the attributes on
the base class in addition to these attributes. Attributes do not need to be
set.
</td></tr>
<tr><td><img src="./images/icons/callouts/3.png" alt="3" /></td><td>
Attributes have both a name ("metadata:cert") as well as a type
("RDFX509Cert"). In this example, the VFSGRRClient object will contain a CERT
attribute which will be an instance of the type RDFX509Cert.
</td></tr>
<tr><td><img src="./images/icons/callouts/4.png" alt="4" /></td><td>
An attribute can also be marked as ready for indexing. This means that
whenever this attribute is updated, the corresponding index is also updated.
</td></tr>
</table></div>
<div class="imageblock">
<div class="content">
<img src="images/aff4_attributes.jpg" alt="images/aff4_attributes.jpg" />
</div>
<div class="title">Figure 2. View of an AFF4 VFSGRRClient with some of its attributes.</div>
</div>
<div class="paragraph"><p>The figure above illustrates an AFF4 Object of type VFSGRRClient. It has a URN
of "aff4:/C.880661da867cfebd". The figure also list all the attributes attached
to this object. Notice how some attributes are listed under the heading
<em>AFF4Object</em> (since they are defined at that level) and some are listed under
<em>VFSGRRClient</em> since they are defined under the VFSGRRClient schema.</p></div>
<div class="paragraph"><p>The figure also gives an <em>Age</em> for each attribute. This is the time when the
attribute was created. Since GRR deals with fluid, constantly changing systems,
each fact about the system must be tagged with the point in time where that fact
was known. For example, in a future time, the hostname may change. In that case
we will have several versions for the HOSTNAME attribute, each correct for that
point in time. We consider the entire object to have a new version when a
versioned attribute changes.</p></div>
<div class="imageblock">
<div class="content">
<img src="images/pslist.jpg" alt="images/pslist.jpg" />
</div>
<div class="title">Figure 3. Example of multiple versions present at the same time.</div>
</div>
<div class="paragraph"><p>The Figure above shows a process listing performed on this client. The view we
currently see shows the the process listing at one point in time, but we can
also see a UI offering to show us previous versions of the same object.</p></div>
<div class="paragraph"><p>AFF4 objects take care of their own serialization and unserialization and the
data store technology is abstracted.  Usually AFF4 objects are managed using the
aff4 FACTORY:</p></div>
<div class="listingblock">
<div class="content">
<pre><tt>In [8]: pslist = aff4.FACTORY.Open("aff4:/C.d74adcb3bef6a388/devices\    <img src="./images/icons/callouts/1.png" alt="1" />
   /memory/pslist", mode="r", age=aff4.ALL_TIMES)

In [9]: pslist                                                           <img src="./images/icons/callouts/2.png" alt="2" />
Out[9]: &lt;AFF4MemoryStream@7F2664442250 = aff4:/C.d74adcb3bef6a388/devices/memory/pslist&gt;

In [10]: print pslist.read(500)                                          <img src="./images/icons/callouts/3.png" alt="3" />
 Offset(V) Offset(P)  Name                 PID    PPID   Thds   Hnds   Time
---------- ---------- -------------------- ------ ------ ------ ------ -------------------
0xfffffa8001530b30 0x6f787b30 System                    4      0     97    520 2012-05-14 18:21:33
0xfffffa80027119d0 0x6e5119d0 smss.exe                256      4      3     33 2012-05-14 18:21:34
0xfffffa8002ce3060 0x6dee3060 csrss.exe               332    324      9    611 2012-05-14 18:22:25
0xfffffa8002c3

In [11]: s = pslist.Get(pslist.Schema.SIZE)                             <img src="./images/icons/callouts/4.png" alt="4" />

In [12]: print type(s)                                                  <img src="./images/icons/callouts/5.png" alt="5" />
&lt;class 'grr.lib.aff4.RDFInteger'&gt;

In [13]: print s                                                        <img src="./images/icons/callouts/6.png" alt="6" />
4938

In [14]: print s.age                                                    <img src="./images/icons/callouts/7.png" alt="7" />
2012-05-21 14:48:20

In [15]: for s in pslist.GetValuesForAttribute(pslist.Schema.SIZE):     <img src="./images/icons/callouts/8.png" alt="8" />
   ....:     print s, s.age
4938 2012-05-21 14:48:20
4832 2012-05-21 14:20:30
4938 2012-05-21 13:53:05</tt></pre>
</div></div>
<div class="colist arabic"><table>
<tr><td><img src="./images/icons/callouts/1.png" alt="1" /></td><td>
We have asked the aff4 factory to open the AFF4 object located at the unique
location of <em>aff4:/C.d74adcb3bef6a388/devices/memory/pslist</em> for reading. The
factory will now go to the data store, and retrieve all the attributes which
comprise this object. We also indicate that we wish to examine all versions of
all attributes on this object.
</td></tr>
<tr><td><img src="./images/icons/callouts/2.png" alt="2" /></td><td>
We receive back an AFF4 object of type <em>AFF4MemoryStream</em>. This is a stream
(i.e. it contains data) which stores all its content in memory.
</td></tr>
<tr><td><img src="./images/icons/callouts/3.png" alt="3" /></td><td>
Since it is a stream, it also implements the stream interface (i.e. supports
reading and seeking). Reading this stead gives back the results from running
Volatility&#8217;s pslist against the memory of the client.
</td></tr>
<tr><td><img src="./images/icons/callouts/4.png" alt="4" /></td><td>
The SIZE attribute is attached to the stream and indicates how much data is
contained in the stream. Using the Get() interface we retrieve the most recent
one.
</td></tr>
<tr><td><img src="./images/icons/callouts/5.png" alt="5" /></td><td>
The attribute is strongly typed, and it is an instance of an RDFInteger.
</td></tr>
<tr><td><img src="./images/icons/callouts/6.png" alt="6" /></td><td>
The RDFInteger is able to stringify itself sensibly.
</td></tr>
<tr><td><img src="./images/icons/callouts/7.png" alt="7" /></td><td>
All attributes carry the timestamp when they were created. The last time the
SIZE attribute was updated was when the object was written to last.
</td></tr>
<tr><td><img src="./images/icons/callouts/8.png" alt="8" /></td><td>
We can now retrieve all versions of this attribute - The pslist Flow was run
on this client 3 times at different dates. Each time the data is different.
</td></tr>
</table></div>
</div>
</div>
<div class="sect1">
<h2 id="_client_path_specifications">Client Path Specifications</h2>
<div class="sectionbody">
<div class="paragraph"><p>One of the nice things about the GRR client is the ability to nest file
readers. For example, we can read files inside an image using the sleuthkit and
also directly through the API. We can read registry keys using REGFI from raw
registry files as well as using the API.  The way this is implemented is using a
pathspec.</p></div>
<div class="sect2">
<h3 id="_pathspecs">Pathspecs</h3>
<div class="paragraph"><p>The GRR client has a number of drivers to virtualize access to different objects
create a Virtual File System (VFS) abstraction. These are called <em>VFS Handlers</em>
and they provide typical file-like operations (e.g. read, seek, tell and stat).
It is possible to recursively apply different drivers in the correct order to
arrive at a certain file like object. In order to specify how drivers should be
applied we use <em>Path Specifications</em> or pathspec.</p></div>
<div class="paragraph"><p>Each VFS handler is constructed from a previous handler and a pathspec. The
pathspec is just a collection of arguments which make sense to the specific VFS
handler. The type of the handler is carried by the pathtype parameter:</p></div>
<div class="dlist"><dl>
<dt class="hdlist1">
pathtype: OS
</dt>
<dd>
<p>
    Implemented by the grr.client.vfs_handlers.file module is a VFS Handler for
    accessing files through the normal operating system APIs.
</p>
</dd>
<dt class="hdlist1">
pathtype: TSK
</dt>
<dd>
<p>
    Implemented by the grr.client.vfs_handlers.sleuthkit module is a VFS Handler
    for accessing files through the sleuthkit. This Handle depends on being
    passed a raw file like object, which is interpreted as the raw device.
</p>
</dd>
<dt class="hdlist1">
pathtype: MEMORY
</dt>
<dd>
<p>
    Implemented by the grr.client.vfs_handlers.memory module is a VFS Handler
    implementing access to the system&#8217;s raw memory. This is used by the
    volatility plugins for memory analysis.
</p>
</dd>
</dl></div>
<div class="paragraph"><p>A pathspec is a list of components. Each component specifies a way to derive a
new python file-like object from the previous file-like object. For example,
image we have the following pathspec:</p></div>
<div class="literalblock">
<div class="content">
<pre><tt>path:   /dev/sda1
pathtype: OS
nested_path: {
   path: /windows/notepad.exe
   pathtype: TSK
}</tt></pre>
</div></div>
<div class="paragraph"><p>This opens the raw device /dev/sda1 using the OS driver. The TSK driver is then
given the previous file like object and the nested pathspec instructing it to
open the /windows/notepad.exe file after parsing the filesystem in the previous
step.</p></div>
<div class="paragraph"><p>This can get more involved, for example:</p></div>
<div class="literalblock">
<div class="content">
<pre><tt>path:   /dev/sda1
pathtype: OS
nested_path: {
   path: /windows/system32/config/system
   pathtype: TSK
   nested_path: {
      path: SOFTWARE/MICROSOFT/WINDOWS/
      pathtype: REGISTRY
  }
}</tt></pre>
</div></div>
<div class="paragraph"><p>Which means to use TSK to open the raw registry file and then REGFI to read the
key from it (note that is needed because you generally cant read the registry
file while the system is running).</p></div>
</div>
<div class="sect2">
<h3 id="_pathspec_transformations">Pathspec transformations</h3>
<div class="paragraph"><p>The pathspec tells the client exactly how to open the required file, by nesting
drivers on the client.  Generally, however, the server has no prior knowledge of
files on the client, therefore the client needs to transform the server request
to the pathspec that makes sense for the client. The following are the
transformations which are applied to the pathspec by the client.</p></div>
<div class="sect3">
<h4 id="_file_case_correction_and_path_separator_correction">File Case Correction and path separator correction</h4>
<div class="paragraph"><p>Some filesystems are not case sensitive (e.g. NTFS). However they do preserve
file cases. This means that the same pathspecs with different case filename will
access the same file on disk. This file however, does have a well defined and
unchanging casing.  The client can correct file casing, e.g.:</p></div>
<div class="literalblock">
<div class="content">
<pre><tt>path: c:\documents and settings\
pathtype: OS</tt></pre>
</div></div>
<div class="paragraph"><p>Is corrected to the normalized form:</p></div>
<div class="literalblock">
<div class="content">
<pre><tt>path: /c/Documents and Settings/
pathtype: OS</tt></pre>
</div></div>
</div>
<div class="sect3">
<h4 id="_filesystem_mount_point_conversions">Filesystem mount point conversions</h4>
<div class="paragraph"><p>Sometimes the server requires to read a particular file from the raw disk using
TSK. However, the server generally does not know where the file physically
exists without finding out the mounted devices and their mount points. This
mapping can only be done on the client at request runtime. When the top level
pathtype is TSK, the client knows that the server intends to read the file
through the raw interface, and therefore converts the pathspec to the correct
form using the mount points information. For example:</p></div>
<div class="literalblock">
<div class="content">
<pre><tt>path: /home/user/hello.txt
pathtype: TSK</tt></pre>
</div></div>
<div class="paragraph"><p>Is converted to:</p></div>
<div class="literalblock">
<div class="content">
<pre><tt>path: /dev/sda2
pathtype: OS
nested_path: {
      path: /user/hello.txt
      pathtype: TSK
}</tt></pre>
</div></div>
</div>
<div class="sect3">
<h4 id="_uuids_versus_classical_device_nodes">UUIDs versus "classical" device nodes</h4>
<div class="paragraph"><p>External disks can easily get re-ordered at start time, so that path specifiers
containing /dev/sd? etc. may not be valid anymore after the last reboot. For
that reason the client will typically replace /dev/sda2 or similar strings with
/dev/disk/by-uuid/[UUID] on Linux or other constructions (e.g. pathtype: uuid)
for all clients.</p></div>
</div>
</div>
<div class="sect2">
<h3 id="_life_of_a_client_pathspec_request">Life of a client pathspec request</h3>
<div class="paragraph"><p>How are the pathspecs sent to the client and how are they related to the aff4
system. The figure below illustrate a typical request - in this case to list a
directory:</p></div>
<div class="olist arabic"><ol class="arabic">
<li>
<p>
A ListDirectory Flow is called with a pathspec of:
</p>
<div class="literalblock">
<div class="content">
<pre><tt>path: c:\docume~1\bob\
pathtype: OS</tt></pre>
</div></div>
</li>
<li>
<p>
The flow sends a request for the client action ListDirectory with the
provided pathspec.
</p>
</li>
<li>
<p>
Client calls VFSOpen(pathspec) which opens the file, and correct the pathspec
to:
</p>
<div class="literalblock">
<div class="content">
<pre><tt>path: c:\Documents and Settings\Bob\
pathtype: OS</tt></pre>
</div></div>
</li>
<li>
<p>
Client returns StatResponse for this directory with the corrected pathspec.
</p>
</li>
<li>
<p>
The client AFF4 object maps the pathspec to an AFF4 hierarchy in the AFF4
space. The server flow converts from client pathspec to the aff4 URN for this
object using the PathspecToURN() API. In this case a mapping is created for
files read through the OS apis under <strong>/fs/os/</strong>. Note that the AFF4 URN created
contains the case corrected - expanded pathspec:
</p>
<div class="literalblock">
<div class="content">
<pre><tt>urn = GRRClient.PathspecToURN(pathspec)
urn = aff4:/C.12345/fs/os/c/Documents and Settings/Bob</tt></pre>
</div></div>
</li>
<li>
<p>
The server now creates this object, and stores the corrected pathspec as a
STAT AFF4 attribute.
</p>
</li>
</ol></div>
<div class="paragraph"><p>Client pathspec conversions can be expensive so the next time the server uses
this AFF4 object for a client request, the server can simply return the client
the corrected pathspec. The corrected pathspec has the LITERAL option enabled
which prevents the client from applying any corrections.</p></div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_foreman">Foreman</h2>
<div class="sectionbody">
<div class="paragraph"><p>The Foreman is a client scheduling service. At a regular intervals (defaults to
every 50 minutes) the client will report in asking if there are Foreman actions
for it. At the time of this check in, the Foreman will be queried to decide if
there are any jobs that match the host, if there are, appropriate flows will be
created for the client. This mechanism is generally used by Hunts to schedule
flows on a large number of clients.</p></div>
<div class="paragraph"><p>The foreman maintains a list of rules, if the rule matches a client when it
checks in, the specified flow will execute on the client. The rules work against
AFF4 attributes allowing for things like "All XP Machines" or "All OSX machines
installed after 01.01.2011".</p></div>
<div class="paragraph"><p>The foreman check in request is a special request made by the client that
communicates with a Well Known Flow (W:Foreman). When the server sees this
request it does the following:</p></div>
<div class="olist arabic"><ol class="arabic">
<li>
<p>
Determines how long since this client did a Foreman check-in.
</p>
</li>
<li>
<p>
Determines the set of that are non-expired and haven&#8217;t previously been checked
 by the client.
</p>
</li>
<li>
<p>
Matches those rules against the client&#8217;s attributes to determine if there is a
 match.
</p>
</li>
<li>
<p>
If there is a match, run the associated flow.
</p>
</li>
</ol></div>
<div class="paragraph"><p>The reason for the separate Foreman check-in request is that the rule matching
can be expensive when you have a lot of clients, so having these less frequent
saves a lot of processing.</p></div>
</div>
</div>
<div class="sect1">
<h2 id="_reporting">Reporting</h2>
<div class="sectionbody">
<div class="paragraph"><p>The current (September 2012) reporting capabilities of GRR are very limited. We
have a small set of reports defined in lib/aff4_objects/reports.py that can be
used and serve as examples for extension. There is no UI for these components
currently, but they can be run from the Console.</p></div>
<div class="listingblock">
<div class="content">
<pre><tt>r = reports.ClientListReport(data_store.ACLToken())                <img src="./images/icons/callouts/1.png" alt="1" />
r.Run()                                                            <img src="./images/icons/callouts/2.png" alt="2" />
r.MailReport("joe@mailinator.com")                                 <img src="./images/icons/callouts/3.png" alt="3" />
open("client_list.csv", "w").write(r.AsCsv())                      <img src="./images/icons/callouts/4.png" alt="4" /></tt></pre>
</div></div>
<div class="colist arabic"><table>
<tr><td><img src="./images/icons/callouts/1.png" alt="1" /></td><td>
Create a report that lists all clients and gives basic information about
them. We pass an empty authorization token.
</td></tr>
<tr><td><img src="./images/icons/callouts/2.png" alt="2" /></td><td>
Run the report. This opens all client objects, so may take some time if you
have a lot of clients.
</td></tr>
<tr><td><img src="./images/icons/callouts/3.png" alt="3" /></td><td>
Mail the output of the report to <a href="mailto:joe@mailinator.com">joe@mailinator.com</a>.
</td></tr>
<tr><td><img src="./images/icons/callouts/4.png" alt="4" /></td><td>
Dump the CSV output of the report to client_list.csv.
</td></tr>
</table></div>
<div class="paragraph"><p>Using these report classes as a basis, it should be relatively easy to extend
the reports to do whatever you need.</p></div>
</div>
</div>
<div class="sect1">
<h2 id="_authorization_and_auditing">Authorization and Auditing</h2>
<div class="sectionbody">
<div class="paragraph"><p>GRR contains support for a full authorization and audit API (even for console
users) and is implemented in an abstraction called a Security Manager.
This Security Manager shipped with GRR (September 2012) does not make use of
these APIs and is open by default. However, a deployment may build their own
Security Manager which implements the authorization semantics they require.</p></div>
<div class="paragraph"><p>This infrastructure is noticeable throughout much of the code, as access to any
data within the system requires the presence of a "token". The token contains
the user information and additionally information about the authorization of the
action. This passing of the token may seem superfluous with the current
implementation, but enables developers to create extensive audit capabilities
and interesting modes of authorization.</p></div>
<div class="paragraph"><p>By default, GRR should use data_store.default_token if one is not provided. To
ease use this variable is automatically populated by the console if --client is
used.</p></div>
<div class="paragraph"><p>Token generation is done using the data_store.ACLToken.</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight 3.1.5
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>token <span style="color: #990000">=</span> access_control<span style="color: #990000">.</span><span style="font-weight: bold"><span style="color: #000000">ACLToken</span></span><span style="color: #990000">()</span>
fd <span style="color: #990000">=</span> aff4<span style="color: #990000">.</span>FACTORY<span style="color: #990000">.</span><span style="font-weight: bold"><span style="color: #000000">Open</span></span><span style="color: #990000">(</span><span style="color: #FF0000">"aff4:/C.12345/"</span><span style="color: #990000">,</span> token<span style="color: #990000">=</span>token<span style="color: #990000">)</span></tt></pre></div></div>
</div>
</div>
<div class="sect1">
<h2 id="_developer_information">Developer Information</h2>
<div class="sectionbody">
<div class="paragraph"><p>The following examples detail how developers can extend the GRR framework for
implementing new features.</p></div>
<div class="sect2">
<h3 id="_adding_flows">Adding Flows.</h3>
<div class="paragraph"><p>The most common task is to add a new flow. The following is an example of a
non-trivial flow from the GRR source code:</p></div>
<div class="listingblock">
<div class="title">A sample flow for analyzing client memory using Volatility (abridged and annotated).</div>
<div class="content"><!-- Generator: GNU source-highlight 3.1.5
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt><span style="font-weight: bold"><span style="color: #0000FF">class</span></span> <span style="font-weight: bold"><span style="color: #000000">AnalyzeClientMemory</span></span><span style="color: #990000">(</span>flow<span style="color: #990000">.</span>GRRFlow<span style="color: #990000">):</span>                   <span style="font-style: italic"><span style="color: #9A1900"># <img src="./images/icons/callouts/1.png" alt="1" /></span></span>
  category <span style="color: #990000">=</span> <span style="color: #FF0000">"/Memory/"</span>                                    <span style="font-style: italic"><span style="color: #9A1900"># <img src="./images/icons/callouts/2.png" alt="2" /></span></span>

  <span style="font-weight: bold"><span style="color: #0000FF">def</span></span> <span style="font-weight: bold"><span style="color: #000000">__init__</span></span><span style="color: #990000">(</span>self<span style="color: #990000">,</span> plugins<span style="color: #990000">=</span><span style="color: #FF0000">"pslist,dlllist,modules"</span><span style="color: #990000">,</span> driver_installer<span style="color: #990000">=</span>None<span style="color: #990000">,</span>
               profile<span style="color: #990000">=</span>None<span style="color: #990000">,</span> <span style="color: #990000">**</span>kwargs<span style="color: #990000">):</span>
    <span style="font-weight: bold"><span style="color: #000000">super</span></span><span style="color: #990000">(</span>AnalyseClientMemory<span style="color: #990000">,</span> self<span style="color: #990000">).</span><span style="font-weight: bold"><span style="color: #000000">__init__</span></span><span style="color: #990000">(**</span>kwargs<span style="color: #990000">)</span>
    self<span style="color: #990000">.</span>plugins <span style="color: #990000">=</span> plugins
    self<span style="color: #990000">.</span>driver_installer <span style="color: #990000">=</span> driver_installer
    self<span style="color: #990000">.</span>profile <span style="color: #990000">=</span> profile

  @flow<span style="color: #990000">.</span><span style="font-weight: bold"><span style="color: #000000">StateHandler</span></span><span style="color: #990000">(</span>next_state<span style="color: #990000">=[</span><span style="color: #FF0000">"RunVolatilityPlugins"</span><span style="color: #990000">])</span>  <span style="font-style: italic"><span style="color: #9A1900"># <img src="./images/icons/callouts/3.png" alt="3" /></span></span>
  <span style="font-weight: bold"><span style="color: #0000FF">def</span></span> <span style="font-weight: bold"><span style="color: #000000">Start</span></span><span style="color: #990000">(</span>self<span style="color: #990000">):</span>
    self<span style="color: #990000">.</span><span style="font-weight: bold"><span style="color: #000000">CallFlow</span></span><span style="color: #990000">(</span><span style="color: #FF0000">"LoadMemoryDriver"</span><span style="color: #990000">,</span> next_state<span style="color: #990000">=</span><span style="color: #FF0000">"RunVolatilityPlugins"</span><span style="color: #990000">,</span>
                  driver_installer<span style="color: #990000">=</span>self<span style="color: #990000">.</span>driver_installer<span style="color: #990000">)</span>

  @flow<span style="color: #990000">.</span><span style="font-weight: bold"><span style="color: #000000">StateHandler</span></span><span style="color: #990000">(</span>next_state<span style="color: #990000">=</span><span style="color: #FF0000">"ProcessVolatilityPlugins"</span><span style="color: #990000">)</span>
  <span style="font-weight: bold"><span style="color: #0000FF">def</span></span> <span style="font-weight: bold"><span style="color: #000000">RunVolatilityPlugins</span></span><span style="color: #990000">(</span>self<span style="color: #990000">,</span> responses<span style="color: #990000">):</span>
    <span style="font-weight: bold"><span style="color: #0000FF">if</span></span> responses<span style="color: #990000">.</span>success<span style="color: #990000">:</span>
      memory_information <span style="color: #990000">=</span> responses<span style="color: #990000">.</span><span style="font-weight: bold"><span style="color: #000000">First</span></span><span style="color: #990000">()</span>

      self<span style="color: #990000">.</span><span style="font-weight: bold"><span style="color: #000000">CallFlow</span></span><span style="color: #990000">(</span><span style="color: #FF0000">"VolatilityPlugins"</span><span style="color: #990000">,</span> plugins<span style="color: #990000">=</span>self<span style="color: #990000">.</span>plugins<span style="color: #990000">,</span>
                    device<span style="color: #990000">=</span>memory_information<span style="color: #990000">.</span>device<span style="color: #990000">,</span> profile<span style="color: #990000">=</span>self<span style="color: #990000">.</span>profile<span style="color: #990000">,</span>
                    next_state<span style="color: #990000">=</span><span style="color: #FF0000">"End"</span><span style="color: #990000">)</span>                      <span style="font-style: italic"><span style="color: #9A1900"># <img src="./images/icons/callouts/4.png" alt="4" /></span></span>
    <span style="font-weight: bold"><span style="color: #0000FF">else</span></span><span style="color: #990000">:</span>
      <span style="font-weight: bold"><span style="color: #0000FF">raise</span></span> flow<span style="color: #990000">.</span><span style="font-weight: bold"><span style="color: #000000">FlowError</span></span><span style="color: #990000">(</span><span style="color: #FF0000">"Failed to Load driver: %s"</span> <span style="color: #990000">%</span> responses<span style="color: #990000">.</span>status<span style="color: #990000">)</span>

  @flow<span style="color: #990000">.</span><span style="font-weight: bold"><span style="color: #000000">StateHandler</span></span><span style="color: #990000">()</span>
  <span style="font-weight: bold"><span style="color: #0000FF">def</span></span> <span style="font-weight: bold"><span style="color: #000000">End</span></span><span style="color: #990000">(</span>self<span style="color: #990000">):</span>
    self<span style="color: #990000">.</span><span style="font-weight: bold"><span style="color: #000000">Notify</span></span><span style="color: #990000">(</span><span style="color: #FF0000">"ViewObject"</span><span style="color: #990000">,</span> self<span style="color: #990000">.</span>device_urn<span style="color: #990000">,</span>
                <span style="color: #FF0000">"Completed execution of volatility plugins."</span><span style="color: #990000">)</span></tt></pre></div></div>
<div class="colist arabic"><table>
<tr><td><img src="./images/icons/callouts/1.png" alt="1" /></td><td>
A flow is simply a class which extends flow.GRRFlow.
</td></tr>
<tr><td><img src="./images/icons/callouts/2.png" alt="2" /></td><td>
By specifying a category, the GUI can use this flow automatically.
</td></tr>
<tr><td><img src="./images/icons/callouts/3.png" alt="3" /></td><td>
A state method is decorated with the <em>StatHandler</em> decorator. This also
specifies all the possible other states that can be transitioned from this
state.
</td></tr>
<tr><td><img src="./images/icons/callouts/4.png" alt="4" /></td><td>
The <em>CallFlow</em> method delegates execution to another flow. The responses
from this flows will be fed back into the End state.
</td></tr>
</table></div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_testing">Testing</h2>
<div class="sectionbody">
<div class="paragraph"><p>We aim to keep GRR well unit tested to ensure we can move quickly with new
changes without unknowingly breaking things. Different pieces of the system
are tested to different levels of rigour, but generally new functionality
should not go in without a test.</p></div>
<div class="sect2">
<h3 id="_running_the_tests">Running the Tests</h3>
<div class="paragraph"><p>To run the tests there are a few dependencies you&#8217;ll need to make sure are
sorted.</p></div>
<div class="paragraph"><p>Testing the admin interface requires the webdriver/selenium framework.
First you need to install the framework and chrome driver, we recommend using:</p></div>
<div class="listingblock">
<div class="content">
<pre><tt>apt-get install python-pip    # if you don't have it already
sudo pip install selenium
sudo apt-get install libnss3 chromium-browser   # dependency of chromedriver
wget https://chromedriver.googlecode.com/files/chromedriver_linux64_26.0.1383.0.zip
unzip chromedriver*
sudo mv chromedriver /usr/bin/</tt></pre>
</div></div>
<div class="paragraph"><p>Use the run_tests.py script to run the unit tests.</p></div>
<div class="listingblock">
<div class="content">
<pre><tt>joe@host:~/dev/grr$ PYTHONPATH="" python ./grr/run_tests.py
        SchedulerTest                            PASSED in   0.31s
        FlowFactoryTest                          PASSED in   0.41s
        NetstatTest                              PASSED in   0.40s
        SQLiteFileTest                           PASSED in   0.50s
        BackwardsCompatibleClientCommsTest       PASSED in   0.91s
        GrrWorkerTest                            PASSED in   0.61s
        AFF4GRRTest                              PASSED in   0.41s
        ChromeHistoryTest                        PASSED in   0.41s
        AccessControlTest                        PASSED in   0.41s
        ...
        TestAdministrativeFlows                  FAILED in   0.41s
        ...
        TestFileCollector                        PASSED in   2.02s
        TestWebHistory                           PASSED in   1.31s

Ran 75 tests in 8.98 sec, 74 tests passed, 1 tests failed.</tt></pre>
</div></div>
<div class="paragraph"><p>To run a specific test name the test class on the command line:</p></div>
<div class="listingblock">
<div class="content">
<pre><tt>joe@host:~/dev/grr$ PYTHONPATH="" python ./grr/run_tests.py TestAdministrativeFlows</tt></pre>
</div></div>
</div>
<div class="sect2">
<h3 id="_testing_the_ui">Testing the UI.</h3>
<div class="paragraph"><p>Note: If you are doing this on a remote machine over SSH you need to make
sure your X display is forwarded for Selenium, it will spawn a Firefox window
during the tests.</p></div>
<div class="paragraph"><p>Then run the runtests_test.py script</p></div>
<div class="listingblock">
<div class="content">
<pre><tt>joe@host:~/dev/grr$ PYTHONPATH="" python ./grr/gui/runtests_test.py</tt></pre>
</div></div>
<div class="paragraph"><p>To run a specific test name the test class on the command line:</p></div>
<div class="listingblock">
<div class="content">
<pre><tt>joe@host:~/dev/grr$ PYTHONPATH="" python ./grr/gui/runtests_test.py TestTimelineView</tt></pre>
</div></div>
<div class="sect3">
<h4 id="_debugging_gui_tests">Debugging GUI Tests</h4>
<div class="paragraph"><p>Selenium will instrument the browser flow, but to figure out why something isn&#8217;t
working it&#8217;s often useful to be able to single step through the Selenium actions.
Add a call to import pdb; pdb.set_trace() into your test. e.g.</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight 3.1.5
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>sel<span style="color: #990000">.</span><span style="font-weight: bold"><span style="color: #000000">click</span></span><span style="color: #990000">(</span><span style="color: #FF0000">"css=tbody tr:first td"</span><span style="color: #990000">)</span>
<span style="font-weight: bold"><span style="color: #000080">import</span></span> pdb<span style="color: #990000">;</span> pdb<span style="color: #990000">.</span><span style="font-weight: bold"><span style="color: #000000">set_trace</span></span><span style="color: #990000">()</span>
sel<span style="color: #990000">.</span><span style="font-weight: bold"><span style="color: #000000">click</span></span><span style="color: #990000">(</span><span style="color: #FF0000">"css=a:contains(\"View details\")"</span><span style="color: #990000">)</span></tt></pre></div></div>
<div class="paragraph"><p>When the test hits this point it will break into a pdb shell for you to step
through.</p></div>
</div>
</div>
<div class="sect2">
<h3 id="_debugging_client_actions">Debugging Client Actions</h3>
<div class="paragraph"><p>Client actions can be run manually without a flow by using the console. The
following example shows using the ExecutePython client action without a flow.</p></div>
<div class="listingblock">
<div class="content"><!-- Generator: GNU source-highlight 3.1.5
by Lorenzo Bettini
http://www.lorenzobettini.it
http://www.gnu.org/software/src-highlite -->
<pre><tt>client_id <span style="color: #990000">=</span> <span style="color: #FF0000">"C.XXXXXXXXXXX"</span>
token <span style="color: #990000">=</span> access_control<span style="color: #990000">.</span><span style="font-weight: bold"><span style="color: #000000">ACLToken</span></span><span style="color: #990000">()</span>
fd <span style="color: #990000">=</span> aff4<span style="color: #990000">.</span>FACTORY<span style="color: #990000">.</span><span style="font-weight: bold"><span style="color: #000000">Open</span></span><span style="color: #990000">(</span><span style="color: #FF0000">"aff4:/config/python_hacks/myfile.py"</span><span style="color: #990000">)</span>
request <span style="color: #990000">=</span> rdfvalue<span style="color: #990000">.</span><span style="font-weight: bold"><span style="color: #000000">ExecutePythonRequest</span></span><span style="color: #990000">(</span>python_code<span style="color: #990000">=</span>fd<span style="color: #990000">.</span><span style="font-weight: bold"><span style="color: #000000">Read</span></span><span style="color: #990000">(</span><span style="color: #993399">100000</span><span style="color: #990000">))</span>
<span style="font-weight: bold"><span style="color: #000000">StartFlowAndWorker</span></span><span style="color: #990000">(</span>client_id<span style="color: #990000">,</span> <span style="color: #FF0000">"ClientAction"</span><span style="color: #990000">,</span> action<span style="color: #990000">=</span><span style="color: #FF0000">"ExecutePython"</span><span style="color: #990000">,</span>
    args<span style="color: #990000">=</span>request<span style="color: #990000">,</span> break_pdb<span style="color: #990000">=</span>False<span style="color: #990000">,</span> save_to<span style="color: #990000">=</span><span style="color: #FF0000">"/tmp/grrdebug"</span><span style="color: #990000">)</span></tt></pre></div></div>
<div class="paragraph"><p>A few points about the above code:
- This runs a worker on a separate queue called DEBUG which means production
  workers won&#8217;t attempt to process them.
- StartFlowAndWorker will create the flow, send the requests to the client, and
  start a worker to process the returned results.
- break_pdb and save_to are special args to the ClientAction flow to help you
  debug interactively.</p></div>
</div>
</div>
</div>
<div class="sect1">
<h2 id="_key_libraries">Key Libraries.</h2>
<div class="sectionbody">
<div class="paragraph"><p>GRR makes use of a number of key open source projects in its implementation.
These include:</p></div>
<div class="ulist"><ul>
<li>
<p>
AFF4 <a href="http://code.google.com/p/aff4/">http://code.google.com/p/aff4/</a>  (datastore uses aff4 design)
</p>
</li>
<li>
<p>
Django <a href="https://www.djangoproject.com/">https://www.djangoproject.com/</a> (admin UI uses django templates)
</p>
</li>
<li>
<p>
Ipython <a href="http://ipython.org/">http://ipython.org/</a>  (Console is built on ipython)
</p>
</li>
<li>
<p>
Jquery <a href="http://jquery.com/">http://jquery.com/</a>  (UI code extensively uses jquery)
</p>
</li>
<li>
<p>
M2Crypto <a href="http://chandlerproject.org/Projects/MeTooCrypto/">http://chandlerproject.org/Projects/MeTooCrypto/</a>  (provides python
 encryption libraries for comms and signing)
</p>
</li>
<li>
<p>
Mongo <a href="http://www.mongodb.org/">http://www.mongodb.org/</a>  (datastore runs on mongo)
</p>
</li>
<li>
<p>
Protobuf <a href="http://code.google.com/p/protobuf/">http://code.google.com/p/protobuf/</a>  (messages and interfaces are
defined as protobufs)
</p>
</li>
<li>
<p>
psutil <a href="http://code.google.com/p/psutil/">http://code.google.com/p/psutil/</a>  (used for native process and network
 data)
</p>
</li>
<li>
<p>
PyTSK <a href="http://code.google.com/p/pytsk/">http://code.google.com/p/pytsk/</a>  (client uses pytsk to talk to tsk)
</p>
</li>
<li>
<p>
Sleuthkit <a href="http://www.sleuthkit.org/">http://www.sleuthkit.org/</a>  (client uses tsk for raw access)
</p>
</li>
<li>
<p>
VerifySigs <a href="http://code.google.com/p/verify-sigs/">http://code.google.com/p/verify-sigs/</a> (used for authenticode
signature handling and verification)
</p>
</li>
<li>
<p>
Volatility <a href="http://code.google.com/p/volatility/">http://code.google.com/p/volatility/</a> (used for memory forensics
capabilities)
</p>
</li>
</ul></div>
</div>
</div>
</div>
<div id="footnotes"><hr /></div>
<div id="footer">
<div id="footer-text">
Last updated 2013-04-04 14:18:08 CEST
</div>
</div>
</body>
</html>
